Skip to content

Conversation

@mcheshkov
Copy link
Contributor

@mcheshkov mcheshkov commented Feb 13, 2025

Check List

  • Tests have been run in packages where changes made if available
  • Linter has been run for changed code
  • Tests for the changes have been added if not covered yet
  • Docs have been added / updated if required

Description of Changes Made (if issue reference is not provided)

Add new member expression: PatchMeasure.
It allows to change aggregation type and add measure filters for existing measure. This is to allow queries like MIN(avgMeasure) and SUM(CASE dim = 'foo' WHEN TRUE THEN sumMeasure ELSE NULL END) in SQL API.

Under the hood

  • SQL API now allows mismatched aggregation functions and aggregation on top of CASE in SQL pushdown rules, and rewrites that to special expression __patch_measure(measure_column, 'new_agg_type', boolean_expression)
  • This expression is syntactic-only, it should never execute, instead it would be handled by wrapper SQL generation
  • Generated member expression is passed to JS-side
  • api-gateway evaluates it from JSON from JS functions
  • BaseQuery and BaseMeasure now can handle new form: generate patched measure definition and use original measure during traversing and collecting

Supporting changes

  • Changed rewrites for usual measures in SQL pushdown from just measure_column to MEASURE(measure_column), to make rewrites uniform
  • Turn member expressions to tagged union in JSON representation

@mcheshkov mcheshkov force-pushed the patch-measure-member-expression branch 2 times, most recently from 011a583 to 8792910 Compare February 13, 2025 01:13
@codecov
Copy link

codecov bot commented Feb 13, 2025

Codecov Report

Attention: Patch coverage is 88.24532% with 138 lines in your changes missing coverage. Please review.

Project coverage is 76.56%. Comparing base (a4fc5ec) to head (d522158).
Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
...t/cubesql/cubesql/src/compile/engine/df/wrapper.rs 82.64% 55 Missing ⚠️
.../cubejs-schema-compiler/src/adapter/BaseMeasure.ts 3.33% 29 Missing ⚠️
...sql/src/compile/rewrite/rules/wrapper/aggregate.rs 95.61% 16 Missing ⚠️
packages/cubejs-api-gateway/src/gateway.ts 38.09% 10 Missing and 3 partials ⚠️
rust/cubesql/cubesql/src/transport/ext.rs 74.46% 12 Missing ⚠️
...t/cubesql/cubesql/src/compile/engine/udf/common.rs 87.75% 6 Missing ⚠️
...es/cubejs-schema-compiler/src/adapter/BaseQuery.js 42.85% 3 Missing and 1 partial ⚠️
packages/cubejs-api-gateway/src/query.js 85.71% 2 Missing ⚠️
rust/cubesql/cubesql/src/compile/rewrite/mod.rs 85.71% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           master    #9218       +/-   ##
===========================================
+ Coverage   47.86%   76.56%   +28.69%     
===========================================
  Files         171      400      +229     
  Lines       21312   105029    +83717     
  Branches     3687     3704       +17     
===========================================
+ Hits        10201    80414    +70213     
- Misses      10676    24179    +13503     
- Partials      435      436        +1     
Flag Coverage Δ
cube-backend 47.80% <33.33%> (-0.06%) ⬇️
cubesql 83.91% <91.83%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mcheshkov mcheshkov force-pushed the patch-measure-member-expression branch from 214a45a to 6275676 Compare February 13, 2025 17:02
if (s.path()) {
return [s.cube().name].concat(this.evaluateSymbolSql(s.path()[0], s.path()[1], s.definition()));
} else if (s.patchedMeasure?.patchedFrom) {
return [s.patchedMeasure.patchedFrom[0]].concat(this.evaluateSymbolSql(s.patchedMeasure.patchedFrom[0], s.patchedMeasure.patchedFrom[1], s.definition()));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's at least add the word Path to the name of the patchedFrom field (ideally, of course, make it a structure with cubeName and Name). Otherwise, every time we come across such a code fragment, we will have to go to the patchedMeasure definition to understand what patchedFrom and its 0th and 1st elements represent

@mcheshkov mcheshkov force-pushed the patch-measure-member-expression branch from 6275676 to 88c5012 Compare March 26, 2025 14:23
@mcheshkov mcheshkov force-pushed the patch-measure-member-expression branch from 88c5012 to 85910bb Compare April 2, 2025 16:02
@mcheshkov mcheshkov marked this pull request as ready for review April 2, 2025 16:38
@mcheshkov mcheshkov requested review from a team as code owners April 2, 2025 16:38
@mcheshkov mcheshkov force-pushed the patch-measure-member-expression branch 2 times, most recently from 94afdf6 to 553c15f Compare April 7, 2025 12:13
@mcheshkov mcheshkov force-pushed the patch-measure-member-expression branch from 553c15f to d522158 Compare April 8, 2025 13:39
@mcheshkov mcheshkov merged commit 128280a into master Apr 8, 2025
78 checks passed
@mcheshkov mcheshkov deleted the patch-measure-member-expression branch April 8, 2025 14:16
marianore-muttdata pushed a commit to MuttData/cube that referenced this pull request Jun 17, 2025
Add new member expression: `PatchMeasure`.
It allows to change aggregation type and add measure filters for existing measure. This is to allow queries like `MIN(avgMeasure)` and `SUM(CASE dim = 'foo' WHEN TRUE THEN sumMeasure ELSE NULL END)` in SQL API.

Under the hood
* SQL API now allows mismatched aggregation functions and aggregation on top of `CASE` in SQL pushdown rules, and rewrites that to special expression `__patch_measure(measure_column, 'new_agg_type', boolean_expression)`
* This expression is syntactic-only, it should never execute, instead it would be handled by wrapper SQL generation
* Generated member expression is passed to JS-side
* `api-gateway` evaluates it from JSON from JS functions
* `BaseQuery` and `BaseMeasure` now can handle new form: generate patched measure definition and use original measure during traversing and collecting

Supporting changes
* Changed rewrites for usual measures in SQL pushdown from just `measure_column` to `MEASURE(measure_column)`, to make rewrites uniform
* Turn member expressions to tagged union in JSON representation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants